The Impact of Item Parameter Estimation on Computerized Adaptive Testing with Item Cloning (CT 02-06) (PDF)

Item cloning techniques can greatly reduce the cost of item writing and enhance the flexibility of item presentation. An important consequence of cloning is that it may cause variability in the item parameters. Recently, Glas and van der Linden (in press, 2005) proposed a multilevel item response model where it is assumed that the item parameters of a 3-parameter logistic (3PL) model or a 3-parameter normal ogive (3PNO) model are sampled from a multivariate normal distribution associated with a parent item. In the sequel, the model will be referred to as the item cloning model, which will be abbreviated ICM. Several procedures for item bank calibration and computerized adaptive testing (CAT) were proposed. The latter procedures were developed under the usual assumption that the item parameters are known. However, in practice, item parameters have to be estimated, which introduces an error component that can have substantial effects. For the standard 3PL model, van der Linden and Glas (2000, 2001) show that capitalization on estimation error can lead to a substantial loss of precision. In the present report, this finding is corroborated for the ICM. It is shown that the problem can be solved by a Bayesian item selection procedure where the uncertainty about the item parameters is taken into account by implicating their posterior distributions. These posterior distributions are generated using the Gibbs Sampler. A simulation study is presented to illustrate the performance of the method. Introduction Item cloning is based on a formal description of a set of parent items and an algorithm to derive a larger set of operational items from them. These parent items have been known as item forms, item templates, or item shells, whereas the items generated from them are now widely known as item clones. Comprehensive reviews of the technology of item cloning are given in Bejar (1993) and Roid and Haladyna (1982). Recently, Glas and van der Linden (in press, 2005) proposed a multilevel item response (IRT) model where it is assumed that the item parameters of a 3PL model are sampled from a multivariate normal distribution associated with a parent item. The model is fully Bayesian in the sense that (informative) priors are formulated for all hyperparameters describing the distributions of the item parameters within the populations. The numerical procedure used to calculate the estimates is the Markov Chain Monte Carlo (MCMC) simulation (Gibbs Sampler). Glas and van der Linden (2005a) proposed several procedures for CAT with item clones. These procedures were developed under the usual assumption that the item parameters are known; however, in practice, item parameters have to be estimated, which introduces an error component that can have substantial effects. For the standard 3PL model, van der Linden and Glas (2000, 2001) show that capitalization on estimation error can lead to a substantial loss of precision. The main cause of this phenomenon is that highly discriminating items (items with a high discrimination parameter in the 3PL or 3PNO model) are both likely to be selected and are bound to have large standard errors. Paradoxically, the size of the item bank is negatively related to the precision of the ability estimates. The reason is that a large item bank contains more highly discriminating items with large standard errors. In the present report, this phenomenon is also found for the ICM. However, it is shown that the problem can be solved by a Bayesian item selection procedure where the uncertainty about the item parameters is taken into account using their posterior distributions generated with the Gibbs Sampler. 1